test(cua-driver): isolate intermittent recorder shutdown - #3739

Refs #3713

Scope

Maintainer-selected diagnostic/TDD workstream, not a product fix or release candidate. Agreed seams: public recording tools, real external encoder process, canonical Windows recording preflight. The three-second shutdown deadline, encoder arguments, and strict video acceptance remain unchanged. Temporary workflow extracts only the canonical preflight prefix, builds fixtures once per runner, preserves every observation separately, and stops that runner on its first failure. Hosted runs only; no privileged self-hosted dispatch.

Evidence

152 total diagnostic observations, none reproduced the failure. These are clustered observations, not 152 independent-machine trials or a reliability guarantee. Recorded stdin operations succeeded; no forced kills. Each completed canonical preflight independently validates recording/media evidence. All recordings, traces, source SHAs, and per-observation receipts are in recorder-observations artifacts (instance suffixes for matrix runs), also retained locally at /Users/administrator/3713-evidence/. No full native behavior matrix was run by these diagnostic workflows; the green summary job is not desktop certification.

Separate prior baseline: main eca08d3 passed native preflight and 43 native cases in 34579403297.

Current implementation and limits

Current head defers shutdown logging until after the wait result; the earlier extra pre-stop try_wait and logging were removed to test instrumentation perturbation. The CPU-pressure code is removed from current head; its tested commit and artifacts remain. Observations still cannot establish the old failure cause: historical logs collapsed nonzero exit and timeout, and no historical child exit trace exists. No evidence supports extending the deadline, bypassing the Chocolatey shim, or weakening video acceptance.

Status / next step

Draft, root cause unresolved, no evidence-backed red regression or product fix. Diagnostic changes must not ship as-is. Another identical warm-loop run is unlikely to be useful: next investigation should recover a failing historical-equivalent environment/executable provenance or capture the next naturally failing native preflight with the shipped #3714 error distinction, then build the smallest deterministic public-tool regression. No retries to relabel a failed gate green. Accidental initial empty-marker dispatch 34596492814 was cancelled and is not evidence. No release claim; no-release label applies only to this diagnostic draft.

Historical-source replay and environment mismatch

A further controlled replay ran the exact original failing main product SHA bd4c10020cd7cac07c0d19b0f53ba4b007fbcb15, without diagnostic product instrumentation, using the draft workflow only as an external preflight runner. Run 34633677564: 60/60 preflights passed across three hosted machines; every log reports one test passed and the recording finalized. Product SHA is retained in every observation receipt. This brings the follow-up to 212 passing diagnostic preflight observations, not a demonstrated repair or a reproducibility rate for independent machines.

New verified confound: original failed job 102947618595 used windows-2025-vs2026 image 20260824.214.3. Recent diagnostics and original-source replay use 20260907.229.1. The original passing main run 34307741070 also used the older image, so the image version alone is not established as causal. Both image manifests report Windows Server build 33296 and Chocolatey 2.7.4; both run logs report FFmpeg 9.0.1. Do not attribute this to an OS-version or FFmpeg-version change based on those values.

The exact old product also passes on the new image, so recent product changes or added tracing are not required for these observed passes. The historical failure still cannot be classified as encoder exit versus timeout. No new failed shutdown was captured. The next useful boundary is historical environment/executable provenance, not another identical warm-loop batch. The old runner-image release exposes metadata/SBOM assets, not a downloadable VM disk; no equivalent retained Windows host was identified in available Fleet inventory. Obtaining a historical-equivalent runner/snapshot or a fresh natural failure with #3714 diagnostics remains the concrete blocker to causal isolation.

No tech spec for a product fix has been generated: user requested that only after isolation. This draft remains diagnostic-only, unready, and not a release claim.

Encoder pipe-lifetime isolation experiment

Current diagnostic head d2285f49e changes the temporary workflow from preflight loops to a small external-process experiment; it does not execute the native behavior matrix. Run 34636737714 ran 20 fixed observations: actual Chocolatey shim versus direct FFmpeg, each with stdin held open versus immediately closed, five per condition. The owned FFmpeg process was suspended before queuing q, then resumed after 100 ms, so it could not consume the shutdown byte before the writer-close decision. This deliberately controls scheduling rather than hoping to hit a race. Same gdigrab/encoding arguments and three-second wait; subprocess stdout is separately captured in this diagnostic, unlike production null stdout. No upstream reference code was copied into the implementation.

All 20 exited normally with code 0 in 289–440 ms after resume. No forced kills. This negative control does not support lost queued shutdown input on immediate pipe close in the tested environment. Stdin-close changes are therefore not justified by this evidence. Raw per-condition MP4s, stderr/stdout, executable hashes, and receipts are retained in recorder-observations-1 and locally under /Users/administrator/3713-evidence/34636737714/.

Additional source trace: preflight drops the sentinel before stopping recording, but the driver-owned child reaper runs only after McpDriver::drop calls stop_e2e_recording. Source inspection does not establish premature daemon teardown as the cause. The original artifact has a finalized moov layout, but that does not prove child process success; historical exit/timeout classification remains missing. Still no reproduced #3713 shutdown failure and no causal tech spec.

Public-tool short-lifetime sweep

At a58a9376e, run 34637929345 exercised the agreed public StartRecordingTool -> StopRecordingTool -> GetRecordingStateTool seam with the actual Windows FFmpeg backend. Two isolated hosted jobs used the Chocolatey shim and the package-owned direct executable respectively. Each performed 500 sequential recording lifecycles, sweeping post-start hold times through 0–200 ms via a fixed modular sequence, stopping immediately on any error and retaining each start/stop/state response and MP4. This targets short-lifetime polling/scheduling phases and same-process reuse rather than repeating the full GUI matrix.

Verified results: 500/500 shim lifecycles and 500/500 direct lifecycles returned successful stops and retained video paths. Maximum public stop latency: shim 675 ms, direct 661 ms. No reproduced shutdown error. These are API/process observations; this sweep does not independently decode every MP4 or certify GUI behavior. Artifacts: recorder-observations-shim and recorder-observations-direct, plus local /Users/administrator/3713-evidence/34637929345/.

Current branch adds an ignored, explicitly invoked diagnostic test at the confirmed public seam. It is not an evidence-backed red regression for #3713 and must not be represented as one. The draft workflow currently runs this sweep instead of native GUI cases. No implementation-ready fix spec has been authored because a failure mechanism remains unverified. Historical missing exit/timeout data cannot be recovered from the retained generic error, MP4 layout, or these successful controls.